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Motives and Goals 
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Motives 

• Yes, it's obvious this is about marketing 

• Any product will probably contain vulnerabilities 

• Awarding dangerous security practices is much 
worse 



Public records give an incomplete picture 
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Goals 

Highlight a product that's an example of this 
problem, and why vulnerability statistics do not 
accurately reflect product security 

Attempt to use publicly available statistics that come 
up with a model that does work 
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Name Nomination 


Choosing A Winner 






Info Security Products Guide Pay for Nomination 


No official public criteria. 




SC Magazine Unknown 


Popular vote 




Techworld.com Unknown 


Unknown 




Information Security Magazine Editor Chosen 


Popular vote 
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Product X and Vendor Y 

Why public statistics aren't a complete picture 
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Product X 



It's a Secret shhh! Hi Lawyers! 

• Provides a web service/interface on a network appliance 
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Product X: Findings 



A manual application security review was performed on the 
device without access to the source code 

The following vulnerabilities were found: 

— Eight high-risk issues 

— Six medium-risk issues 

— Nine low-risk issues 
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Product X: Serious Findings 



This is a subset of the High and Medium risk issues found: 

• Systemic Cross-Site Scripting 

Almost any variable was vulnerable, including variables stored by the 
application (Persistent Cross-Site Scripting) 

• Privilege Escalation 

Browser-supplied user ID while in a valid session could be changed, 
using an easily predictable method, for privilege escalation. 

• Custom Web Server 

Re-inventing the wheel and introducing bugs such as arbitrary system 
file access, including the password file. 
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Product X: Serious Findings (Cont.) 

Session Hijacking 

Poor implementation resulted in users able to steal sessions of users 
logging in around the same time of day. 

• Custom, Weak Session ID Algorithm 

Without getting into details that would give it away: 
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Product X: Reaction 



So What? 
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Vendor Y 



Major software vendor 

• Two independently discovered vulnerabilities, medium or 
higher 

One occurs on their own servers (still) 



Vendor Response: *Crickets* 
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Lies, Statistics, and Awards 
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What Awards Really Mean 



Problems with gathering statistics 

• FUD 

• Sources 

• Lack of History 
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Sample Statistics 



Methodology 

Three Categories 

• Two Awards 

• Competitors 

• Variety of Sources 
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W ® Awards: Anti-Malware 


















Award Product 


Highs 


Mediums 


Lows 


SC Magazine Symantec End-Point Protection 


4 








Info Security Products Guide CoreTrace - Bouncer 4.0 











Nod32 Anti-Virus 


2 


1 


2 


Proventia Network Scanner 











Radware Defense Pro 











Vipre 











Websense 


1 


2 


1 
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*<T Awards: Endpoint Security 




















Award Product Highs 


Mediums Lows 






SC Magazine McAffee Security Center 2 





1 




Info Security Products Guide Parity v4.0. 1 










Checkpoint for Endpoint Security 


2 


2 


Cisco NAC 1 1 


F5 Firepass Remote Access Solutions 5 


2 


18 


Symantec Endpoint Protection 
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#^ Awards: IPSec/SSLVPN 
























Award Product 


Highs 


Mediums Lows 






SC Magazine Cisco ASA 5500 


3 





4 




Info Security Products Guide NCP Secure Enterprise Solution 








2 




Checkpoint Connectra 








2 


Citrix Access Gateway 


1 


1 





F5 Firepass Remote Access Solutions 


5 


2 


17 


Stonesoft StonegateVPN 
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What Awards Really Mean 
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What Awards Really Mean 



Awards Are Marketing 

• Unclear 

• Too Many 

• Press Releases 

• Pointless 
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Better Ways 
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Better Ways 



Credible Award Requirements 

• Open Process 

• Established Products 

• Audit Product Patch Process 

• Relevant Criteria 
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Better Ways 



Alternative Evaluation Criteria 

• References 

• History of Security 

• Talk to Developers 
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STrustwave- 



